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Software  is  increasingly  important  to  the  development  of  effective  netiuork-centric  Department  of  Defense 
combat  systems.  Next-generation  combat  systems  such  as  total  ship  computing  environments,  coordinated 
unmanned  air  vehicle  systems,  and  national  missile  defense  will  use  many  geographically  dispersed  sensors, 
provide  on-demand  situational  awareness  and  actuation  capabilities  for  human  operators,  and  respond 
flexibly  to  unanticipated  run-time  conditions.  These  combat  systems  will  also  increasingly  run  unobtru¬ 
sively  and  autonomously,  shielding  operators  from  unnecessary  details  while  communicating  and  respond¬ 
ing  to  mission-critical  information  at  an  accelerated  operational  tempo.  In  such  environments,  it  is  hard 
to  predict  system  configurations  or  workloads.  This  article  describes  how  adaptive  and  reflective  middle¬ 
ware  systems  (ARMS)  are  being  developed  to  bridge  the  gap  betiveen  military  application  programs  and 
the  underlying  operating  systems  and  communication  software  in  order  to  provide  reusable  services  whose 
qualities  are  critical  to  network-centric  combat  systems.  ARMS  software  can  adapt  in  response  to  dynam¬ 
ically  changing  conditions  for  the  purpose  of  utilizing  the  available  computer  and  communication  resources 
to  the  highest  degree  possible  in  support  of  mission  needs. 


New  and  planned  Department  of 
Defense  (DoD)  combat  systems  are 
inherently  network-centric,  distributed 
real-time  and  embedded  (DRE)  systems  of 
systems.  Combat  systems  have  historically 
been  developed  via  multiple  technology 
bases,  where  each  system  brings  its  own 
networks,  computers,  displays,  software, 
and  people  to  maintain  and  operate  it. 
Unfortunately,  not  only  are  these  stove¬ 
pipe  architectures  proprietary,  but  by 
tightly  coupling  many  functional  and 
quality  of  service  (QoS)  aspects  they 
impede  these  DRE  system  features: 

•  Assurability  is  needed  to  guarantee  effi¬ 
cient,  predictable,  scalable,  and 
dependable  QoS  from  sensors  to 
shooters. 

•  Adaptability  is  needed  to  (re)configure 
combat  systems  dynamically  to  sup¬ 
port  varying  workloads  or  missions 
over  their  life  cycles. 

•  Affordability  is  needed  to  reduce  initial 
nonrecurring  combat  system  acquisi¬ 
tion  costs  and  recurring  upgrade  and 
evolution  costs. 

In  recognition  of  the  importance  of 
enhancing  affordability,  recent  DoD  pro¬ 
grams  such  as  the  Aegis  destroyer  program 
[1],  the  New  Attack  Submarine  program 
[2],  the  Weapons  Systems  Open 
Architecture  program  [3],  and  the 
Unmanned  Combat  Air  Vehicle  (UCAV) 
program  [4]  have  adopted  strong  open  sys¬ 
tems  approaches  to  system  design  and 
commercial-off-the-shelf  (COTS)  refresh 
strategies.  Ultimately,  open  systems 
approaches  are  more  likely  to  be  robust 
with  respect  to  change  over  the  long  life 
cycles  typical  of  military  systems.  For 
example,  the  affordability  of  certain  types 


of  DoD  systems  such  as  logistics  and  mis¬ 
sion  planning  have  been  improved  by 
using  COTS  technologies. 

However,  many  of  today’s  procure¬ 
ment  efforts  aimed  at  integrating  COTS 
into  mission-critical  DRE  combat  systems 
have  largely  failed  to  support  life-cycle 
affordability,  assurability,  and  adaptability 
effectively  since  they  focus  mainly  on  ini¬ 
tial  nonrecurring  acquisition  costs  and  do 
not  reduce  recurring  software  life-cycle 
costs,  such  as  COTS  refresh  and  subset¬ 
ting  combat  systems  for  foreign  military 
sales  [5] .  Likewise,  many  COTS  products 
lack  support  for  controlling  key  QoS 
properties  such  as  predictable  latency,  jit¬ 
ter,  and  throughput;  scalability;  depend¬ 
ability;  and  security.  The  inability  to  con¬ 
trol  these  QoS  properties  with  sufficient 
confidence  compromises  combat  system 
adaptability  and  assurability,  e.g.,  a  pertur¬ 
bation  in  the  behavior  of  a  COTS  product 
that  would  be  acceptable  in  commercial 
applications  could  lead  to  loss  of  life  and 
property  in  military  applications. 

Historically,  conventional  COTS  soft¬ 
ware  has  been  unsuitable  for  use  in  mis¬ 
sion-critical  DRE  combat  systems  due  to 
either  of  the  following: 

•  It  is  flexible  and  standard,  but  inca¬ 
pable  of  guaranteeing  stringent  QoS 
demands,  which  restricts  assurability. 

•  It  is  partially  QoS-enabled,  but  inflex¬ 
ible  and  non-standard,  which  restricts 
adaptability  and  affordability. 

As  a  result,  the  rapid  progress  in 
COTS  software  for  mainstream  business 
information  technology  (IT)  has  not  yet 
become  as  broadly  applicable  for  mission- 
critical  DRE  combat  systems.  Until  this 
problem  is  resolved  effectively,  DRE  sys¬ 


tem  integrators  and  warfighters  will  not  be 
able  to  take  advantage  of  future  advances 
in  COTS  software  in  a  dependable,  time¬ 
ly,  and  cost  effective  manner.  Developing 
the  new  generation  of  assurable,  adapt¬ 
able,  and  affordable  COTS  software  tech¬ 
nologies  is  therefore  essential  for  U.S. 
national  security. 

Although  the  near-term  use  of  COTS 
software  in  DRE  systems  will  be  limited  in 
scope  and  domain,  the  prospects  for  the 
longer  term  are  much  brighter.  Given  the 
proper  advanced  research  and  develop¬ 
ment  (R&D)  context  and  an  effective 
process  for  transitioning  R&D  results,  the 
COTS  market  can  adapt,  adopt,  and 
implement  the  types  of  robust  hardware 
and  software  capabilities  needed  for  mili¬ 
tary  applications.  This  process  takes  a 
good  deal  of  time  to  get  right  and  be 
accepted  by  user  communities,  and  a  good 
deal  of  patience  to  stay  the  course.  When 
successful,  however,  this  process  results  in 
standards  that  codify  the  best-of-breed 
practices  and  technologies  and  the  patterns 
and frameworks  that  reify  the  knowledge  of 
how  to  apply  these  practices  and  technolo¬ 
gies. 

Key  Technical  Challenges  and 
Solutions 

Today’s  economic  and  organizational  con¬ 
straints  -  along  with  increasingly  complex 
requirements  and  competitive  pressures  - 
make  it  infeasible  to  build  complex  dis¬ 
tributed  real-time  system  software  entirely 
from  scratch.  It  has  long  been  accepted 
that  the  use  of  commercial  operating  sys¬ 
tems  and  communication  support  soft¬ 
ware  is  cost-effective  for  all  but  the  most 
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resource-constrained  DRE  systems.  In¬ 
creasingly,  this  same  logic  is  being  applied 
to  middleware,  which  is  reusable  serv¬ 
ice/protocol  component  and  framework 
software  that  services  end-to-end  and  aggre¬ 
gate  combat  systems’  needs  [6].  Middle¬ 
ware  bridges  the  gap  between  these  areas: 

•  Application-level  requirements  and 
mission  doctrine. 

•  The  lower-level,  underlying,  localized 
viewpoints  of  the  operating  systems 
and  communications  support  mecha¬ 
nisms. 

From  the  application  perspective, 
when  middleware  and  the  services  it  con¬ 
stitutes  are  combined  with  traditional  net¬ 
work  and  operating  system  components,  it 
forms  the  new  infrastructure  for  develop¬ 
ing  modern  network-centric  combat  sys¬ 
tems.  In  both  commercial  and  military 
systems,  middleware  performs  functions 
that  are  essential  to  meeting  application- 
level  requirements.  In  military  systems, 
moreover,  the  qualities  of  the  services  pro¬ 
vided  by  the  middleware  are  critical  to  the 
qualities  of  service  that  are  presented  to 
the  end  users  -  the  warfighters. 

Thus,  there  is  a  pressing  need  to  devel¬ 
op,  validate,  and  ultimately  standardize  a 
new  generation  of  adaptive  and  reflective 
middleware  systems  (ARMS)  technologies 
that  will  be  readily  available  and  able  to 
support  stringent  combat  system  func¬ 
tionality  and  QoS  requirements.  Some  of 
the  most  challenging  computing  and  com¬ 
munication  requirements  for  new  and 
planned  DoD  combat  systems  can  be 
characterized  as  follows: 

•  Multiple  QoS  properties  must  be  satis¬ 
fied  in  real-time. 

•  Different  levels  of  service  are  appropri¬ 
ate  under  different  configurations, 
environmental  conditions,  and  costs. 

•  The  levels  of  service  in  one  dimension 
must  be  coordinated  with  and/or  trad¬ 
ed  off  against  the  levels  of  service  in 
other  dimensions  to  meet  mission 
needs,  e.g.,  the  security  and  depend¬ 
ability  of  message  transmission  must 
be  traded  off  against  latency  and  pre¬ 
dictability. 

•  The  need  for  autonomous  and  time- 
critical  application  behavior  necessi¬ 
tates  a  flexible  distributed  system  sub¬ 
strate  that  can  adapt  robustly  to 
dynamic  changes  in  mission  require¬ 
ments  and  environmental  conditions. 
Adaptive  middleware  [3]  is  software 

whose  functional  and  QoS-related  proper¬ 
ties  can  be  modified  in  either  of  these  ways: 

•  Statically,  e.g.,  to  reduce  footprint, 
leverage  capabilities  that  exist  in  spe¬ 
cific  platforms,  enable  functional  sub¬ 


setting,  and  minimize  hardware  and 
software  infrastructure  dependencies. 

•  Dynamically,  e.g.,  to  optimize  system 
responses  to  changing  environments 
or  requirements,  such  as  changing 
component  interconnections,  power- 
levels,  CPU/network  bandwidth, 
latency/jitter,  and  dependability 
needs. 

In  DRE  combat  systems,  adaptive 
middleware  must  make  these  modifica¬ 
tions  dependably,  i.e.,  while  meeting  strin¬ 
gent  end-to-end  QoS  requirements. 

Reflective  middleware  [7]  goes  a  step 
further  in  providing  the  means  for  exam¬ 
ining  the  capabilities  it  offers  while  the 
system  is  running,  thereby  enabling  auto¬ 
mated  adjustment  for  optimizing  those 
capabilities.  Thus,  reflective  middleware 
supports  more  advanced  adaptive  behav¬ 
ior,  i.e.,  the  necessary  adaptations  can  be 
performed  autonomously  based  on  condi¬ 
tions  within  the  system,  in  the  system’s 
environment,  or  in  combat  system  doc¬ 
trine  defined  by  operators  and  administra¬ 
tors. 

Middleware  Structure 
and  Functionality 

Networking  protocol  stacks  can  be  decom¬ 
posed  into  multiple  layers  such  as  the 
physical,  data-link,  network,  transport, 
session,  presentation,  and  application  lay¬ 
ers.  Similarly,  middleware  can  be  decom¬ 
posed  into  multiple  layers  such  as  those 
shown  in  Figure  1. 

We  describe  each  of  these  middleware 
layers  and  outline  some  of  the  COTS  tech¬ 
nologies  in  each  layer  that  are  suitable  (or 
are  becoming  suitable)  to  meet  the  strin¬ 
gent  QoS  demands  of  DRE  combat  sys¬ 
tems. 


Host  Infrastructure  Middleware 

Host  infrastructure  middleware  encapsu¬ 
lates  and  enhances  native  operating  system 
communication  and  concurrency  mecha¬ 
nisms  to  create  portable  and  reusable  net¬ 
work  programming  components  such  as 
reactors,  acceptor-connectors,  monitor 
objects,  active  objects,  and  component 
configurations  [8].  These  components 
abstract  away  the  accidental  incompatibil¬ 
ities  of  individual  operating  systems  and 
help  eliminate  many  tedious,  error-prone, 
and  non-portable  aspects  of  developing 
and  maintaining  networked  applications 
via  low-level  operating  system  program¬ 
ming  application  program  interfaces 
(APIs),  such  as  Sockets  or  POSIX 
Pthreads.  Examples  of  COTS  host  infra¬ 
structure  middleware  that  are  relevant  for 
DRE  combat  systems  include  the  follow¬ 
ing: 

•  The  Adaptive  Communication  Envi¬ 
ronment  (ACE)  [9]  is  a  portable  and 
efficient  toolkit  that  encapsulates  native 
operating  system  network  program¬ 
ming  capabilities  such  as  inter-process 
communication,  static  and  dynamic 
configuration  of  application  compo¬ 
nents,  and  synchronization.  ACE  has 
been  used  in  a  wide  range  of  DoD  DRE 
systems,  including  missile  control, 
avionics  mission  computing,  software 
defined  radios,  and  radar  systems. 

•  Real-Time  Java  Virtual  Machines  imple¬ 
ment  the  Real-Time  Specification  for 
Java  (RTSJ)  [10].  The  RTSJ  is  a  set  of 
extensions  to  Java  that  provide  a  largely 
platform-independent  way  of  executing 
code  by  encapsulating  the  differences 
between  real-time  operating  systems 
and  CPU  architectures.  The  key  fea¬ 
tures  of  RTSJ  deal  with  memory  man- 


Figure  1 :  Middleware  Layers  and  Their  Surrounding  Context 
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agement  and  concurrency.  Although 
RTSJ  implementations  are  still  in 
their  infancy,  they  have  generated 
tremendous  interest  in  the  DoD 
R&D  and  integrator  communities 
due  to  their  potential  for  reducing 
software  development  and  evolution 
costs  significantly. 

Distribution  Middleware 

Distribution  middleware  defines  a  higher- 
level  distributed  programming  model 
whose  reusable  application  program  inter¬ 
faces  and  mechanisms  automate  and 
extend  the  native  operating  system  net¬ 
work  programming  capabilities  encapsu¬ 
lated  by  host  infrastructure  middleware. 
Distribution  middleware  enables  develop¬ 
ers  to  program  distributed  applications 
much  like  stand-alone  applications,  i.e., 
by  invoking  operations  on  target  objects 
or  distributed  components. 

At  the  heart  of  distribution  middle¬ 
ware  are  QoS-enabled  object  request  bro¬ 
kers,  such  as  the  Object  Management 
Group’s  (OMG)  Common  Object 
Request  Broker  Architecture  (CORBA) 
[4,  11].  CORBA  is  distribution  middle¬ 
ware  that  allows  objects  to  interoperate 
across  networks  without  hard-coding 
dependencies  on  their  location,  program¬ 
ming  language,  operating  system  plat¬ 
form,  communication  protocols  and 
interconnects,  and  hardware  characteris¬ 
tics.  In  1998  the  OMG  adopted  the  Real- 
Time  CORBA  specification  [12],  which 
extends  CORBA  with  features  that  allow 
DRE  applications  to  reserve  and  manage 
CPU,  memory,  and  networking  resources. 
Real-Time  CORBA  implementations 
have  been  used  in  dozens  of  DoD  combat 
systems,  including  avionics  mission  com¬ 
puting  [4],  submarine  combat  control  sys¬ 
tems  [13],  signal  intelligence  and 
Command,  Control,  Communications, 
Computers,  Intelligence,  Surveillance, 
and  Reconnaissance  systems,  software 
defined  radios,  and  radar  systems. 

Common  Middleware  Services 

Common  middleware  services  augment 
distribution  middleware  by  defining  high¬ 
er-level,  domain-independent,  reusable 
services  that  have  proven  necessary  in 
most  distributed  application  contexts  to 
deal  with  multi-computer  environments 
effectively.  In  addition,  these  services  pro¬ 
vide  components  that  allow  application 
developers  to  concentrate  on  program¬ 
ming  application  logic,  without  the  need 
to  write  the  plumbing  code  needed  to 
develop  distributed  applications  using 
lower  level  middleware  features  directly. 
For  example,  whereas  distribution 


middleware  focuses  largely  on  managing 
end-system  resources  in  support  of  an 
object-oriented  distributed  programming 
model,  common  middleware  services 
focus  on  allocating,  scheduling,  and  coor¬ 
dinating  various  end-to-end  resources 
throughout  a  distributed  system  using  a 
component  programming  and  scripting 
model.  Developers  can  reuse  these  servic¬ 
es  to  manage  global  resources  and  per¬ 
form  recurring  distribution  tasks  that 
would  otherwise  be  re-implemented  by 
each  application  or  integrator. 

Examples  of  common  middleware 
services  include  the  OMG’s 
CORBAServices  [14]  and  the  CORBA 
Component  Model  (CCM)  [15],  which 
provide  domain-independent  interfaces 

“Today’s  economic 
and  organizational 
constraints  -  along 
with  increasingly 
complex  requirements 
and  competitive 
pressures  -  make  it 
infeasible  to  build 
complex  distributed 
real-time  system 
software  entirely  from 
scratch.” 


and  distribution  capabilities  that  can  be 
used  by  many  distributed  applications. 
The  OMG  CORBAServices  and  CCM 
specifications  define  a  wide  variety  of 
these  services,  including  event  notifica¬ 
tion,  naming,  security,  and  fault  toler¬ 
ance.  Not  all  of  these  standard  services  are 
sufficiently  refined  today  to  be  usable  off 
the  shelf  for  DRE  combat  systems. 
However,  the  form  and  content  of  these 
common  middleware  services  will  contin¬ 
ue  to  mature  and  evolve  to  meet  the 
expanding  requirements  of  DRE. 

Domain-Specific  Middleware 
Services 

Domain-specific  middleware  services  are 
tailored  to  the  requirements  of  particular 
combat  system  domains,  such  as  avionics 
mission  computing,  radar  processing, 
weapons  targeting,  or  command  and  deci¬ 
sion  systems.  Unlike  the  previous  three 
middleware  layers  -  which  provide  broad¬ 
ly  reusable  horizontal  mechanisms  and 
services  -  domain-specific  middleware 


services  are  targeted  at  vertical  market  seg¬ 
ments.  From  a  COTS  perspective, 
domain-specific  services  are  the  least 
mature  of  the  middleware  layers  today. 
This  immaturity  is  due  in  part  to  the  his¬ 
torical  lack  of  distribution  middleware 
and  common  middleware  service  stan¬ 
dards,  which  are  needed  to  provide  a  sta¬ 
ble  base  upon  which  to  create  domain- 
specific  middleware  services.  Since  they 
embody  knowledge  of  a  domain,  however, 
domain-specific  middleware  services  have 
the  most  potential  to  increase  the  quality 
and  decrease  the  cycle  time  and  effort  that 
DoD  integrators  require  to  develop  par¬ 
ticular  classes  of  DRE  combat  systems. 

A  mature  example  of  domain-specific 
middleware  services  appears  in  the  Boeing 
Bold  Stroke  architecture  [4].  Bold  Stroke 
uses  COTS  hardware  and  middleware  to 
produce  a  non-proprietary,  standards- 
based  component  architecture  for  military 
avionics  mission  computing  capabilities, 
such  as  navigation,  data  link  manage¬ 
ment,  and  weapons  control.  A  driving 
objective  of  Bold  Stroke  was  to  support 
reusable  product-line  applications,  lead¬ 
ing  to  a  highly  configurable  application 
component  model  and  supporting  mid¬ 
dleware  services.  The  domain-specific 
middleware  services  in  Bold  Stroke  are 
layered  upon  common  middleware  servic¬ 
es  (the  CORBA  Event  Service),  distribu¬ 
tion  middleware  (Real-Time  CORBA  and 
the  tactical  air  operations  object  request 
broker  [16]),  and  infrastructure  middle¬ 
ware  advanced  computing  environment, 
and  have  been  demonstrated  to  be  highly 
portable  for  different  COTS  operating 
systems  (e.g.,  VxWorks),  interconnects 
(e.g.,  VME),  and  processors  (e.g., 
PowerPC). 

Recent  Progress 

Significant  progress  has  occurred  during 
the  last  five  years  in  DRE  middleware 
research,  development,  and  deployment 
within  the  DoD,  stemming  in  large  part 
from  the  following  trends: 

Maturation  of  Standards 

During  the  past  decade,  middleware  stan¬ 
dards  have  been  established  and  have 
matured  considerably  with  respect  to 
DRE  requirements.  For  example,  the 
OMG  has  recently  adopted  the  following 
DRE-related  specifications: 

•  Minimum  CORBA  removes  non- 
essential  features  from  the  full  OMG 
CORBA  specification  to  reduce  foot¬ 
print  so  that  CORBA  can  be  used  in 
memory-constrained  embedded  sys¬ 
tems. 
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•  Real-Time  CORBA  includes  features 
that  allow  applications  to  reserve  and 
manage  network,  CPU,  and  memory 
resources  predictably  end  to  end. 

•  CORBA  Messaging  exports  additional 
QoS  policies  such  as  timeouts,  request 
priorities,  and  queuing  disciplines  to 
applications. 

•  Fault-Tolerant  CORBA  uses  entity 
redundancy  of  objects  to  support 
replication,  fault  detection,  and  fail¬ 
ure  recovery. 

Robust  and  interoperable  implemen¬ 
tations  of  these  CORBA  capabilities  and 
services  are  now  available  from  multiple 
vendors.  Moreover,  emerging  standards 
such  as  Dynamic  Scheduling  Real-Time 
CORBA,  Real-Time  CORBA  publish- 
subscribe  services,  the  Real-Time 
Specification  for  Java,  and  the  Distributed 
Real-Time  Specification  for  Java  are 
extending  the  scope  of  open  standards  for 
a  wider  range  of  DoD  applications. 

Dissemination  of  Patterns, 
Frameworks 

A  substantial  amount  of  R&D  effort  dur¬ 
ing  the  past  decade  has  also  focused  on 
the  following  means  of  promoting  the 
development  and  reuse  of  high  quality 
middleware  technology: 

•  Patterns  codify  design  expertise  that 
provides  time-proven  solutions  to 
commonly  occurring  software  prob¬ 
lems  that  arise  in  particular  contexts 
[17].  Patterns  can  simplify  the  design, 
construction,  and  performance  tun¬ 
ing  of  DRE  applications  by  codifying 
the  accumulated  expertise  of  develop¬ 
ers,  architects,  and  systems  engineers 
who  have  already  confronted  similar 
problems  successfully. 

•  Frameworks  are  concrete  realizations 
of  related  patterns  [18]  that  provide 
an  integrated  set  of  components  that 
collaborate  to  provide  a  reusable 
architecture  for  a  family  of  related 
applications.  Middleware  frameworks 
include  strategized  selection  and  opti¬ 
mization  patterns  so  that  multiple, 
independently  developed  capabilities 
can  be  integrated  and  configured 
automatically  to  meet  the  functional 
and  QoS  requirements  of  particular 
DRE  applications. 

Historically,  the  knowledge  required 
to  develop  predictable,  scalable,  efficient, 
and  dependable  mission-critical  DoD 
DRE  combat  systems  has  existed  largely 
in  programming  folklore,  the  heads  of 
experienced  researchers  and  developers,  or 
buried  deep  within  millions  of  lines  of 
complex  source  code.  Moreover,  docu¬ 


menting  complex  systems  with  today’s 
popular  software  modeling  methods  and 
tools,  such  as  the  Unified  Modeling 
Language  (UML),  only  capture  how  a  sys¬ 
tem  is  designed,  but  do  not  necessarily 
articulate  why  a  system  is  designed  in  a 
particular  way,  which  complicates  subse¬ 
quent  software  evolution  and  optimiza¬ 
tion. 

Middleware  patterns  and  frameworks 
help  address  these  problems  by  systemati¬ 
cally  capturing  combat  system  design 
expertise  in  a  readily  accessible  and 
reusable  format,  thereby  raising  the  level 
at  which  systems  engineers  and  applica¬ 
tion  developers  approach  the  decision 
making  and  implementation  of  their  sys¬ 
tems.  Two  efforts  to  provide  suitable  guid- 


“  Given  the  proper 
advanced  R&D  context 
and  an  effective  process 
for  transitioning  R&D 
results,  the  COTS  market 
can  adapt,  adopt,  and 
implement  the  types  of ... 
capabilities  needed  for 
military  applications  ...” 


ance  for  the  development  of  military  sys¬ 
tems  are  the  New  Attack  Submarine 
(NAS)  [2]  and  the  Aegis  Shipbuilding 
Program.  NAS  developed  a  guidance  doc¬ 
ument  detailing  allowable  standards  for 
the  NAS  C3I  system,  and  the  Aegis  pro¬ 
gram  developed  a  guidance  document  for 
Baseline  7  phase  I  [19].  These  documents 
were  instrumental  in  guiding  the  design 
of  these  systems. 

Much  of  the  pioneering  R&D  on 
middleware  patterns,  frameworks,  and 
standards  for  DRE  combat  systems  has 
been  conducted  as  part  of  the  Defense 
Advanced  Research  Projects  Agency’s 
(DARPA)  Information  Technology  Office 
Quorum  Program  [20],  which  played  a 
leading  role  in  the  following: 

•  Demonstrating  the  viability  of  host 
infrastructure  middleware  and  distri¬ 
bution  middleware  for  DoD  combat 
systems  by  providing  the  foundation 
for  managing  key  QoS  attributes  such 
as  real  time  behavior,  dependability, 
and  system  survivability  from  a  net¬ 
work-centric  middleware  perspective. 

•  Transitioning  a  number  of  new  mid¬ 


dleware  perspectives  and  capabilities 
into  DoD  acquisition  programs  [4, 
21]  and  commercially  supported 
products. 

•  Establishing  the  technical  viability  of 
collections  of  systems  that  can 
dynamically  adapt  [3]  their  collective 
behavior  to  varying  operating  condi¬ 
tions,  in  service  of  delivering  the 
appropriate  application  level  response 
under  these  different  conditions. 

The  Quorum  program  focused  heavi¬ 
ly  on  CORBA  open  systems  middleware 
and  yielded  many  results  that  transitioned 
into  standardized  service  definitions  and 
implementations  for  the  Real-Time  [4] 
and  Fault-Tolerant  [22]  CORBA  specifi¬ 
cation  and  production.  Quorum  is  an 
example  of  how  a  focused  government 
R&D  effort  can  leverage  its  results  by 
exporting  them  into,  and  combining 
them  with,  other  on-going  public  and  pri¬ 
vate  activities  by  using  a  common  open 
middleware  substrate.  Prior  to  the  viabili¬ 
ty  of  standards-based  COTS  middleware 
platforms,  these  same  R&D  results  would 
have  been  buried  within  custom  or  pro¬ 
prietary  systems,  serving  only  as  an  exis¬ 
tence  proof,  rather  than  as  the  basis  for 
realigning  the  DoD  R&D  and  integrator 
communities. 

Successful  DoD  technology  transition 
most  often  results  from  a  partnership 
between  technology  developers  and  tech¬ 
nology  users.  One  of  the  most  successful 
examples  of  such  partnerships  is  the  joint 
DARPA/Aegis  High  Performance  Distrib¬ 
uted  Computing  program  (HiPer-D). 
Through  the  use  of  prototyping  and  sys¬ 
tem-scale  experiments,  this  program  has 
demonstrated  the  effectiveness  of  a  num¬ 
ber  of  DARPA  and  standards-based 
COTS  technologies  for  building  DRE 
combat  systems  that  are  efficient,  scalable, 
fault  tolerant,  and  flexible  in  their  design 
and  operation. 

Looking  Ahead 

Due  to  advances  in  COTS  technologies 
outlined  earlier,  host  infrastructure  mid¬ 
dleware  and  distribution  middleware  have 
now  been  demonstrated  and  deployed  in  a 
number  of  mission-critical  DRE  combat 
systems.  Since  off-the-shelf  middleware 
technology  has  not  yet  matured  to  cover 
the  realm  of  large-scale  dynamically 
changing  systems,  however,  COTS  DRE 
middleware  has  been  applied  to  relatively 
small-scale  and  statically  configured 
embedded  systems.  To  satisfy  the  highly 
application-  and  mission-specific  QoS 
requirements  in  network-centric  system- 
to-  system  environments ,  DRE  middleware 
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Figure  2:  Decoupling  Functional  and  QoS  Attribute  Paths  in  QuO 


must  therefore  be  enhanced  to  support 
common  and  domain-specific  middle¬ 
ware  services  that  can  manage  the  follow¬ 
ing  resources  effectively: 

•  Communication  bandwidth,  e.g.,  net¬ 
work  level  status  information  and 
management  services,  scalability  to 
102  subnets  and  103  nodes,  and 
dynamic  connections  with  controlled 
and  reserved  bandwidth  to  enhance 
real-time  predictability. 

•  Distributed  real-time  scheduling  and 
allocation  of  DRE  system  artifacts 
(such  as  CPUs,  networks,  UAVs,  mis¬ 
siles,  torpedoes,  radar,  illuminators, 
etc),  e.g.,  fast  and  predictable  behavior 
of  widely  dispersed  components  that 
use  the  managed  communication  capa¬ 
bilities  and  bandwidth  reservations. 

•  Distributed  system  dependability,  e.g., 
policy-based  selection  of  replication 
options  to  control  footprint  and  reac¬ 
tive  behavior  to  failures. 

•  Distributed  system  security,  e.g., 
dynamically  variable  object  access  con¬ 
trol  policies  and  effective,  combined 
real-time  dependability,  and  security 
interactions. 

Ironically,  there  is  little  or  no  scientific 
underpinning  for  QoS-enabled  resource 
management,  despite  the  demand  for  it  in 
most  distributed  systems  [23].  Today’s  sys¬ 
tem  designers  develop  concrete  plans  for 
creating  global,  end-to-end  functionality. 
These  plans  contain  high-level  abstractions 
and  doctrine  associated  with  resource  man¬ 
agement  algorithms,  relationships  between 
these,  and  operations  upon  these.  There  are 
few  techniques  and  tools,  however  that 
enable  users,  i.e.,  commanders,  administra¬ 
tors,  and  operators,  and  developers,  i.e., 
systems  engineers  and  application  design¬ 
ers,  and/or  applications  to  express  such 
plans  systematically,  reason  about  and 


refine  them,  and  have  these  plans  enforced 
automatically  to  manage  resources  at  mul¬ 
tiple  levels  in  network-centric  combat  sys¬ 
tems. 

To  address  this  problem,  the  R&D 
community  needs  to  discover  and  set  the 
technical  approach  that  can  significantly 
improve  the  effective  utilization  of  net¬ 
works  and  end-systems  that  DRE  combat 
systems  depend  upon  by  creating  middle¬ 
ware  and  distributed  resource  management 
technologies  and  tools  that  can  automati¬ 
cally  allocate,  schedule,  control,  and  opti¬ 
mize  customizable  -  yet  standards-compli- 
ant  and  verifiably  correct  -  software-inten¬ 
sive  systems.  To  promote  a  common  tech¬ 
nology  base,  the  interfaces  and  (where 
appropriate)  the  protocols  used  by  the 
middleware  should  be  based  on  established 
or  emerging  industry  or  DoD  standards 
that  are  relevant  for  DRE  combat  systems. 
However,  the  protocol  and  service  imple¬ 
mentations  should  be  customizable  -  stati¬ 
cally  and  dynamically  -  for  specific  DoD 
DRE  combat  system  requirements. 

To  achieve  these  goals,  middleware 
technologies  and  tools  need  to  be  based 
upon  some  type  of  layered  architecture 
along  with  QoS  adaptive  middleware  serv¬ 
ices  such  as  the  one  shown  in  Figure  2  and 
based  on  empirical  investigations  of  this 
type  of  capability  [3] . 

The  Quality  Objects  (QuO)  [24]  proj¬ 
ect  is  an  example  of  such  a  layered  archi¬ 
tecture  designed  to  manage  and  package 
adaptive  QoS  capabilities  as  common  mid¬ 
dleware  services.  The  QuO  architecture 
decouples  DRE  middleware  and  applica¬ 
tions  along  the  following  two  dimensions: 

•  Functional  paths  are  flows  of  informa¬ 
tion  between  client  and  remote  server 
applications.  In  distributed  systems, 
middleware  ensures  that  this  informa¬ 
tion  is  exchanged  efficiently,  pre¬ 


dictably,  dependably,  and  securely 
between  remote  peers,  and  in  a  manner 
that  scales  well  to  large  configurations. 
The  information  itself  is  largely  appli¬ 
cation-specific  and  determined  by  the 
functionality  being  provided  (hence 
the  term  functional  path). 

•  QoS  attribute  paths  are  responsible  for 
determining  how  well  the  functional 
interactions  behave  end  to  end  with 
respect  to  key  DRE  system  QoS  prop¬ 
erties  such  as  the  following: 

1 .  How  and  when  resources  are  com¬ 
mitted  to  client/server  interactions 
at  multiple  levels  of  distributed  sys¬ 
tems. 

2.  The  proper  application  and  system 
behavior  if  available  resources  are 
less  than  the  expected  resources. 

3.  The  failure  detection  and  recovery 
strategies  necessary  to  meet  end-to- 
end  dependability  requirements 
under  anomalous  conditions. 

In  next-generation  combat  systems,  the 
middleware  -  rather  than  operating  sys¬ 
tems  or  networks  in  isolation  —  will  be 
responsible  for  separating  DRE  system 
QoS  attribute  properties  from  the  func¬ 
tional  application  properties.  Middleware 
will  also  coordinate  the  QoS  of  various 
DRE  system  and  application  resources  end 
to  end.  The  architecture  in  Figure  2 
enables  these  properties  and  resources  to 
change  independently,  e.g.,  over  different 
distributed  system  configurations  for  the 
same  application. 

The  architecture  in  Figure  2  is  based 
on  the  expectation  that  QoS  attribute 
paths  will  be  developed,  configured,  moni¬ 
tored,  managed,  and  controlled  by  a  differ¬ 
ent  set  of  specialists  (such  as  systems  engi¬ 
neers,  administrators,  operators,  and  per¬ 
haps  someday  automated  agents)  and  tools 
than  those  customarily  responsible  for  pro¬ 
gramming  functional  paths  in  DRE  sys¬ 
tems.  The  middleware  is  therefore  respon¬ 
sible  for  collecting,  organizing,  and  dissem¬ 
inating  QoS-related  meta-information  that 
is  needed  to  do  the  following: 

•  Monitor  and  manage  how  well  the 
functional  interactions  occur  at  multi¬ 
ple  levels  of  DRE  systems. 

•  Enable  the  adaptive  and  reflective  deci¬ 
sion-making  needed  to  support  QoS 
attribute  properties  robustly  in  the  face 
of  rapidly  changing  mission  require¬ 
ments  and  environmental  conditions. 
Researching  and  developing  these  mid¬ 
dleware  capabilities  is  crucial  to  ensure  that 
the  aggregate  behavior  of  future  network¬ 
centric  combat  systems  is  dependable, 
despite  local  failures,  transient  overloads, 
and  dynamic  functional  or  QoS  reconfigu¬ 
rations. 

To  simultaneously  enhance  assurability, 
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adaptability,  and  affordability,  the  middle¬ 
ware  techniques  and  tools  developed  in 
future  R&D  programs  increasingly  need  to 
be  application-independent,  yet  customiz¬ 
able  within  the  interfaces  specified  by  a 
range  of  open  standards  such  as  these: 

•  The  OMG  Real-Time  CORBA  specifi¬ 
cations  and  The  Open  Group’s  QoS 
Task  Force. 

•  The  Java  Expert  Group  Real-Time 
Specification  for  Java  (RTSJ)  and  the 
emerging  Distributed  RTSJ. 

•  The  IEEE  Real-Time  Portable  Oper¬ 
ating  System  (POSIX)  specification. 

Conclusions 

As  a  result  of  much  previous  R&D  and 
transition  experience,  network-centric  sys¬ 
tems  today  are  constructed  as  a  series  of  lay¬ 
ers  of  intertwined  technical  capabilities  and 
innovations.  The  main  emphasis  at  the 
lower  layers  is  in  providing  the  core  com¬ 
puting  and  communication  resources  and 
services  that  drive  network-centric  comput¬ 
ing:  the  individual  computers,  the  net¬ 
works,  and  the  operating  systems  that  con¬ 
trol  the  individual  host  and  the  message 
level  communication. 

At  the  upper  layers,  various  types  of 
middleware  are  starting  to  bridge  the  previ¬ 
ously  formidable  gap  between  the  lower- 
level  resources  and  services  and  the  abstrac¬ 
tions  that  are  needed  to  program,  organize, 
and  control  systems  composed  of  coordi¬ 
nated,  rather  than  isolated,  components. 
Key  capabilities  in  the  upper  layers  include 
common  and  domain-specific  middleware 
services  that  provide  the  following: 

•  Enforcing  real-time  behavior  across 
computational  nodes. 

•  Managing  redundancy  across  elements 
to  support  dependable  computing. 

•  Controlling  end-to-end  adaptive  behav¬ 
ior  in  responding  to  changes  in  operat¬ 
ing  conditions  while  continuing  to 
meet  application  needs. 

These  new  middleware  services  make 
the  coordinated  use  of  multiple  computing 
elements  feasible  and  affordable  by  control¬ 
ling  the  hardware,  network,  and  end-sys¬ 
tem  mechanisms  that  affect  mission,  sys¬ 
tem,  and  application  QoS  delivery  and 
tradeoffs  that  are  needed  to  deliver  the  right 
QoS  at  the  right  time  under  the  prevailing 
conditions. 

Adaptive  and  reflective  middleware  sys¬ 
tems  (ARMS)  is  a  key  emerging  paradigm 
that  will  help  to  simplify  the  development, 
optimization,  validation,  and  integration  of 
DRE  middleware  in  DoD  combat  systems. 
In  particular,  ARMS  will  allow  researchers 
and  system  integrators  to  develop  and 
evolve  complex  combat  systems  assurably, 
adaptively,  and  affordably  through  the  fol¬ 


lowing: 

•  Devising  optimizers,  meta-program¬ 
ming  techniques,  and  multi-level  dis¬ 
tributed  dynamic  resource  manage¬ 
ment  protocols  and  services  for  ARMS 
that  will  enable  DoD  DRE  systems  to 
configure  standard  COTS  interfaces 
without  the  penalties  incurred  by 
today’s  conventional  COTS  software 
product  implementations.  Many  net¬ 
work-centric  DoD  combat  systems 
require  these  DRE  middleware  capabil¬ 
ities. 

•  Standardizing  COTS  at  the  middleware 
level,  rather  than  just  at  lower  hard¬ 
ware/networks/operating  system  levels. 
The  primary  economic  benefits  of  mid¬ 
dleware  stem  from  extending  standard¬ 
ization  up  several  levels  of  abstraction 
so  that  DRE  middleware  technology  is 
readily  available  for  COTS  acquisition 
and  customization. 

As  COTS  implementations  of  middle¬ 
ware  standards  mature  in  their  functional 
quality  and  QoS,  they  are  helping  to  lower 
the  total  ownership  costs  of  combat  sys¬ 
tems.  For  example,  Real-Time  and  Fault- 
Tolerant  CORBA  implementations  are  cre¬ 
ating  a  common  base  of  COTS  technology 
that  enables  complex  DRE  middleware 
capabilities  to  be  reconfigured  and  reused, 
rather  than  reinvented  repeatedly  or 
reworked  from  proprietary  stovepipe  archi¬ 
tectures  that  are  inflexible  and  expensive  to 
maintain,  evolve,  and  optimize.  Additional 
information  on  middleware  for  DRE  systems 
is  available  at  <www.ece.uci.edu/-schmidt/ 
TAO.html>> 
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